LOS ANGELES POLICE COMMISSION 


TRAINING EVALUATION AND 
MANAGEMENT SYSTEM (TEAMS) II 
AUDIT, PHASE // 

(Fiscal Year 2007-2008) 


Conducted by the 


OFFICE OF THE INSPECTOR GENERAL 


ANDRE BIROTTE, JR. 
Inspector General 


June 6, 2008 





OFFICE OF THE INSPECTOR GENERAL 
FISCAL YEAR 2007/2008 
TRAINING EVALUATION AND MANAGEMENT SYSTEM (TEAMS) IT 
AUDIT, PHASE II 


PURPOSE 


Pursuant to Consent Decree Paragraph 137, the Office of the Inspector General (OIG) conducted 
a Training Evaluation and Management System (TEAMS) II Audit (Audit). Specifically, 
Paragraph 137 requires the Inspector General to audit the quality and timeliness of the Los 
Angeles Police Department’ s (LAPD or Department) use of TEAMS II and to evaluate the 
Department’s compliance with Consent Decree Paragraph 47. 


BACKGROUND 


This is the second TEAMS II Audit conducted by the OIG. The first TEAMS II Audit 

(Phase I), issued in November 2007, primarily focused on system-generated action items (AIs).' 
This second TEAMS II Audit (Phase II) primarily focused on supervisor-generated action items 
(SAIs). 


Certain subparagraphs (a, b, c, e, h, 1 and m) of Consent Decree Paragraph 47 relate to SAIs, and 
generally require the following: 


e SAIs shall be generated for both sworn officers and supervisors for annual performance 
evaluations, pay-grade advancements and promotions, employee’ transfers, and periodic 
monitoring purposes. 

e The supervisors generating the SAIs shall review the subject employee’s recent work history 
for possible “at-risk” patterns, using TEAMS II and other information, with the results of this 
review adequately and timely documented within TEAMS II. 

e An intensive review must take place if it appears an employee is exhibiting “at-risk” 
behavior, and any actions taken should be based on all relevant and appropriate information, 
not solely on the number and percentages of incidents recorded in TEAMS IL. 

e Appropriate follow-up, managerial or supervisory actions (including non-disciplinary actions 
or no action), shall be taken based on the results of the SAI reviews. 


In addition to the Consent Decree, the primary document governing Als and SAIs has been 
Special Order No. 22, 2007 (dated July 12, 2007). This Special Order provided Department 
supervisors and managers with information on how to complete AIs and SAIs. It also provided 
details as to the various disposition types (e.g., “Comment Card,” “Training,” “Commendation,” 
“No Action,” etc.) that could be utilized, and when each disposition type should be used. 


The OIG generally considers SAIs a lower risk than Als because an AI is automatically 
generated when an employee has an inordinate amount of uses of force, complaints, vehicle 
pursuits, preventable traffic collisions, or claims/lawsuits in comparison to his/her peer group. 
Indeed, the Department’s implementation of the AI process is the “heart” of the early warning 
system required by the Consent Decree. It allows Department supervisors and managers to 
review an employee’s activity to evaluate whether the employee is exhibiting any type of 





' That Audit also articulated disputed data correction and employee access to their TEAMS II record. 
? Although the OIG uses the term “employee,” it is referring to only sworn personnel of the Department. 


Training Evaluation and Management System (TEAMS) IT Audit, Phase II 
Page 2 of 9 
1.0 


potential “‘at-risk” behavior that should be addressed or monitored. In contrast, a SAI is more of 
a routine process in which Department supervisors or managers will review an employee’s 
TEAMS II record? when an employee transfers, promotes, advances or has an annual 
performance evaluation. An SAI may also be generated at the discretion of an employee’s 
supervisor for monitoring purposes. 


Since approximately March 2006, the new and updated TEAMS II has been implemented by the 
Department. At that time, and for approximately a year thereafter, the Department’s primary 
focus was to ensure that Als were appropriately being handled by supervisors and managers, 
given that they are generally higher risk and more deserving of the Department’s attention. More 
recently, the Department focused its attention to SAIs. Therefore, the OIG elected to perform 
this Audit of SAIs to assess the Department’s progress within this area. 


SCOPE AND METHODOLOGY 


Audit Objectives 


The Audit’s primary objectives (Objectives), and the related subparagraphs of Consent Decree 
Paragraph 47, were as follows: 


1. Determine if SAIs were generated when required (per Special Order No. 22, 2007) for annual 
employee performance evaluations, and for employee transfers, promotions, and pay-grade 
advancements. ({47(a)(c) and (m)) 


2. Determine if the dispositions, including “No Action,” for SAIs were adequately justified in 
the SAI narratives. Per the Consent Decree, if potential “at-risk” behavior is noted, a more 
intensive review of the employee’s performance should take place. Any actions taken must 
be based on all relevant and appropriate information, and not solely on the number and 
percentages of incidents in any category recorded in TEAMS II. (147(b) and (h)) 


3. Determine if SAIs with a final disposition of other than “No Action” contained evidence that 
necessary actions were taken and appeared on the employees’ TEAMS II record. (47(e)) 


4. Determine if supervisors completed/closed SAIs timely. (147(1)) 





? A TEAMS II record tracks an employee’s work history (e.g., an employee’s complaints, uses of force, vehicle 
pursuits, preventable traffic collisions, claims/lawsuits, commendations, work assignments, training, etc.). 
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Populations and Samples 


For Objective Nos. 1 and 4, sampling was not applicable, as the OIG wanted to analyze the entire 
population of SAIs generated (Objective No. 1) and SAIs open (Objective No. 4). For Objective 
Nos. 2 and 3, the OIG selected two samples of SAIs; and for each sample, the size was 
determined by using a one-tail confidence level of 95 percent, an expected error rate of six 
percent, and a plus-precision of seven percent. 


The first sample, used to evaluate Objective No. 2, was selected from a population of 1,169 SAIs 
initiated from June 24 through November 10 of 2007 (Deployment Period Nos. 7 through 11) 
that were completed/closed as of January 3, 2008. From this population, the OIG selected a 
sample of 36 SAIs, stratified by the reason for the SAI4 


The second sample, used to evaluate Objective No. 3, was selected from a population of 69 SAIs 
also initiated from June 24 through November 10 of 2007 (Deployment Period Nos. 7 through 
11) that were completed/closed as of January 3, 2008 with a disposition other than “Cancel,” 
“No Action,” or “No Action — Note for Annual Review Only.” From this population, the OIG 
selected a sample of 24 SAIs, stratified by the disposition of the SAI.” 


OVERALL RESULTS OF THE AUDIT 


The OIG found that SAIs were generally not being generated when required. Furthermore, when 
they were being generated, the assigned final disposition in preparing to close out the SAI 

(e.g., “Comment Card,” “Training,” “Commendation,” “No Action,” etc.) was generally not 
adequately justified by the supervisors and managers. To comply with Department policy, the 
OIG would have expected a narrative in the SAI explaining the reason why the particular final 
disposition was selected. However, the Audit did find that for all of the applicable SAIs, there 
was evidence that the prescribed follow-up action was taken. Furthermore, for 36 SAIs, the OIG 
reviewed each employee’s TEAMS II record for any indication of “at-risk” behavior, and based 
on that review, no significant concerns were identified. 


The OIG provided the Commanding Officer of TEAMS II Development Bureau a draft version 
of this Audit Report, and the Commanding Officer provided a written response. The 
Commanding Officer of TEAMS II Development Bureau advised that she concurred with, and 
had already been aware of, most of the issues raised in this Audit, and accordingly had been 
working over the past several months to modify the system. One major system modification 





4 The original sample of 36 SAIs consisted of 17 “Annual Performance Evaluation” SAIs, seven “Other” SAIs, 
three “Organizational Assessment” SAIs, three “Promotion” SAIs, two “New Assignment” SAIs, two 
“Monitoring” SAIs, one “Transfer” SAI, and one “Special Evaluation Unit” SAI. However, the final sample for 
Objective No. 2 was 30 SAIs, after excluding five SAIs that were generated in error or for testing purposes and one 
SAI for a civilian. Nevertheless, the TEAMS II records for all 36 officers were reviewed for any indication of “at- 
risk” behavior. 

` The sample contained SAIs with eight “Comment Card” dispositions, seven “Commendation” dispositions, four 
“Special Evaluation Report” dispositions, three “Informal Meeting” dispositions, one “Notice to Correct 
Deficiencies” disposition, and one “Complaint” disposition. 
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already implemented Department-wide is that action items for transfers/promotions and annual 
performance evaluations are now automatically generated by TEAMS II, instead of generated 
manually by supervisors. 


DETAILED RESULTS OF THE AUDIT 
Objective No. 1 — Determine if SAIs Were Generated When Required 


Background 

Per Special Order No. 22, 2007, an SAI shall be generated to document a supervisor’s review of 
TEAMS II information for: 1) annual performance evaluations; 2) pay-grade 
advancements/promotions; 3) employee transfers to new assignments; or, 4) periodic monitoring. 
Thus, an important objective in this Audit was to determine if SAIs were generated when 
required. 


Regarding annual performance evaluations, based on LAPD’s deployment of over 9,000 sworn 
personnel, one would expect that over 9,000 SAIs would be generated annually in order to 
comply with Special Order No. 22, 2007. To assess whether SAIs were being generated, the 
OIG requested Personnel Division to provide a monthly listing of annual performance 
evaluations due from July 23 through November 10 of 2007 (Deployment Period Nos. 8 through 
11). For this time period, there were approximately 3,000 annual performance evaluations that 
were due; hence, the OIG expected that approximately 3,000 SAIs would have been generated 
for them. 


Regarding advancements, promotions and transfers, the OIG evaluated the Department’ s 
Transfer Orders from July 23 through November 10 of 2007 (Deployment Period Nos. 8 through 
11) and determined there were over 800 employee pay-grade changes and transfers, and 100 
employee promotions. Hence, the OIG expected that over 900 SAIs would have been generated 
during this time period for them. 


Accordingly, to evaluate whether supervisors were generating SAIs when required, the OIG 
requested TEAMS II Development Bureau to provide a listing of all SAIs, sorted by reason, 
generated between July 23 and November 10 of 2007 (Deployment Period Nos. 8 through 11). 


Results 

In comparing the quantities of SAIs actually generated to the aforementioned quantities of 
expected SAIs, it was noted that supervisors did not generate SAIs for approximately: 

e 81 percent of the expected employee annual performance evaluations due (2419/3000); 
e 93 percent of employee pay-grade changes and transfers (781/836); and, 

e 57 percent of employee promotions (57/100). 
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Actions Taken by TEAMS II Development Bureau 
TEAMS II Development Bureau indicated that, effective April 16, 2008, Transfer Action Items 


(TAIs) have been automatically generated for transfers and promotions. These TAIs contain a 
link to the Transfer Evaluation Report (TER) form that is required to be completed upon an 
employee’s transfer. 


TEAMS II Development Bureau also indicated that, effective June 1, 2008, Performance 
Evaluation Report Action Items (PERs) have been automatically generated for annual 
performance evaluations. SAIs will still be available if a supervisor wants to use them as a 
management tool. Special Order 22, 2007 will be amended to include the TAIs and PERs as 
mandatory, and the SAIs as an available tool. 


Objective No. 2 — Determine if SAI Dispositions Were Adequately Justified 


Background 
Per Special Order No. 22, 2007, in the Investigative Narrative section of the SAI, a supervisor 


must document the extent of his/her inquiry and provide justification for the course of action 
taken, if any, or justification for the decision to take no action. 


Results 
The audit sample of 30 SAIs consisted of 16 SAIs generated for annual performance evaluations 
and 14 SAIs generated for “other” reasons (besides annual performance evaluations). 


For the 16 SAIs generated for annual performance evaluations, the OIG did not expect to find 
detailed narratives in TEAMS II justifying the disposition (which was “No Action” or “No Action 
— Note for Annual Review Only” for 15 of the 16 SAIs). Since detailed narratives are already 
required (per Special Order 47, 2003) to be part of the Performance Evaluation Reports, it would 
be redundant to also have these same narratives in TEAMS II. Nevertheless, in an attempt to 
ensure that the dispositions for these 16 SAIs were appropriate, the OIG reviewed each 
employee’s TEAMS II record for any indication of “at-risk” behavior, and based on that review, 
no significant concerns were identified. 


For the 14 SAIs generated for “other” reasons, the supervisors did not provide justification for 
the disposition of seven (50%) of them. As noted above, Department policy requires a 
supervisor to “document the extent of his/her inquiry and provide justification for the course of 
action taken, if any, or justification for the decision to take no action.” Regarding these seven 
exceptions, in the narrative and/or comment sections of the Action Item History Report, there 
was either no or minimal verbiage to support the assigned final disposition (e.g., “Comment 
Card,” “Training,” “Commendation,” “No Action,” etc.), with the reviewer sometimes just 
restating the reason for, or disposition of, the SAI. To meet the Special Order requirement, the 
OIG would have expected something to the effect of: “The SAI for this employee was assigned a 
disposition of because....” However, understandably, the OIG would generally expect 
less “disposition justification” verbiage for SAIs than for AIs, as SAIs are more routine and 
generally lower risk. 
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In an attempt to ensure that the dispositions for these 14 “other” SAIs were appropriate, the OIG 
reviewed each employee’s TEAMS II record for any indication of “at-risk” behavior, and based 
on that review, no significant concerns were identified. 


Actions Taken by TEAMS II Development Bureau 
TEAMS II Development Bureau indicated that for the new automated TAIs, the only available 


disposition is “Complete,” with no narrative necessary. The supervisor’s narrative regarding the 
transfer can be found on the TER form. Should a supervisor find cause for concern that requires 
further action, he/she will be directed to generate a separate SAI to complete a more detailed 
review and select an appropriate action/disposition. 


TEAMS II Development Bureau also indicated that for the new automated PERs, the only 
available disposition is “No Action — Note for Annual Review Only.” The new PERs have pre-set 
questions in the narrative box that a supervisor must address. Again, should a supervisor find 
cause for concern that requires further action, he/she should generate a separate SAI to complete 
a more detailed review and select an appropriate action/disposition. 


Objective No. 3 — Determine if Prescribed Follow-up Actions for SAIs Were Taken 


Background 
Consent Decree subparagraph 47(e) prescribes that managerial or supervisory actions (including 


non-disciplinary action) should be taken when necessary based on their reviews of TEAMS II 
information. The disciplinary actions specified in the protocol (Special Order No. 22, 2007) 
include: “Informal Meeting;” “Training;” “Special Evaluation Reports;” “Modified Field 
Duties;” “Assigned to Non-Field Duties;” “Risk Management Executive Committee Referral;” 
“Directed Behavioral Science Services Referral;” “Comment Card;” “Notice to Correct 
Deficiencies;” or, “Complaint.” The only non-disciplinary action specified in the protocol is 
“Commendation.” 


Results 

For all 17 of the applicable SAIs,° there was evidence that the prescribed follow-up action was 
taken. However, in reviewing the SAIs, the OIG found that a majority of the SAIs did not 
include documentary evidence that the supervisor met with the employee to discuss the SAI. Per 
Special Order No. 22, 2007, for all dispositions other than “No Action” and “Cancel,” the 
supervisor must meet with the employee to provide an opportunity to review the SAI, and the 
date and time of the meeting must be documented in the SAI. 





° From the original sample of 24 SAIs, the OIG excluded one SAI for a civilian and six SAIs for which the 
appropriate disposition should have been “Cancel,” “No Action,” or “No Action — Note for Annual Review Only.” 
The OIG concluded this based on examination of the Action Item History Reports and the officer TEAMS II reports, 
as well as discussions with the supervisors generating the SAIs regarding their intent. If the correct disposition had 
been assigned for each of these six SAIs, there would have been no required follow-up action. 
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Actions Taken by TEAMS II Development Bureau 
TEAMS II Development Bureau indicated that the TAIs and PERs (and Als and SAIs) include a 


text box named “Date Supervisor Met with the Employee” that must be completed by the 
supervisor before the action item can be completed. Additionally, TEAMS II Development 
Bureau is considering automatically generating a separate follow-up Action Item to the 
supervisor if anything other than “No Action” or “No Action — Note for Annual Review” is 
selected. For example, if a supervisor selects “Training” as a disposition for any type of Action 
Item, the supervisor will receive a follow-up Action Item, requiring the supervisor to note when 
the training was completed and what type of training was provided. 


Objective No. 4 — Determine if Supervisors Completed/Closed SAIs Timely 


Background 
The date that a SAI shall reasonably be completed or finalized by is not dependent on when the 


SAI was generated. Instead, the SAI due date is dependent on the underlying annual 
performance evaluation due date or Transfer Order effective date. Thus, it would be very time 
consuming to test a large sample of completed SAIs by comparing the completion date to the due 
date, as the OIG would have to obtain all the underlying annual performance evaluation due 
dates and Transfer Orders. Furthermore, as noted above, SAIs were not generated for the 
majority of annual performance evaluations and transfers/promotions/advancements. Hence, 
instead of testing completed/closed SAIs for timeliness, the OIG elected to analyze open 
(pending) SAIs to determine their quantity and “age” (how many days open). 


Results 

As of January 22, 2008 (the audit testing date), there were 623 SAIs still open and pending 
completion and finalization. Of these 623 SAIs, 58 were more than 60 days old and 30 were 
more than 120 days old, ranging to 173 days old. While the OIG recognizes that SAIs have 
generally been a lower risk concern than Als, it appears that the Department needs to emphasize 
to supervisors that SAIs should be completed and closed in a timely manner. The OIG provided 
the Commanding Officer of TEAMS II Development Bureau with the details regarding these 58 
old open (pending) SAIs and 12 instances of multiple SAIs generated at the same time for the 
same employee. 





Actions Taken by TEAMS II Development Bureau 
TEAMS II Development Bureau indicated that they are publishing regular reports to each Area 


and Bureau so they can track the completion of all types of action items. 
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Additional Matter — Use of Certain Dispositions for SAIs 


Background 
Per Special Order No. 22, 2007, when preparing to close an SAI, a supervisor must choose from 


one or more of the following dispositions listed in the “Conclusion/ Recommendation’ section of 
the SAI: “Cancel;” “No Action;” “No Action — Note for Annual Review Only;” 
“Commendation; ” “Informal Meeting;” “Training;” “Special Evaluation Reports;” “Modified 
Field Duties;” “Assigned to Non-Field Duties;” “Risk Management Executive Committee 
Referral;” “Directed Behavioral Science Services Referral;” “Comment Card;” “Notice to 
Correct Deficiencies (NTC);” or, “Complaint.” 


The Special Order explains, in detail, when/why/how each of the 14 disposition types are to be 
used. The Special Order further states that all action items receiving a disposition of “Cancel” 
or “No Action” will not be displayed on an employee's TEAMS II record. All action items 

receiving a disposition of “No Action — Note for Annual Review Only” will be displayed on an 
employee's TEAMS II record for a minimum of one year. All action items receiving any other 
disposition will be displayed on an employee's TEAMS II record for a minimum of three years. 


Results 

For the two samples used to evaluate Objective Nos. 2 and 3, there were a total of 55 different 
SAIs. In reviewing the narratives provided for the dispositions (and via correspondence for 
some SAIs), we noted that 17 of the SAIs had been assigned an incorrect disposition by the 
supervisor. The dispositions included nine “No Action,” four “Special Evaluation Report, ”8 one 
“Informal Meeting, ”? one “Commendation,” one “No Action — Note for Annual Review Only,” 
and one “Complaint.”'° The correct disposition for these 17 SAIs should have been “No Action 
— Note for Annual Review Only” for nine, “Cancel” for six, and “No Action” for two. 


It is important to assign the correct disposition because it determines: 

e What, if any, action (corrective or commendation) needs to be taken for/by the employee. 

e If, and/or for how long, the disposition will appear on the employee’s TEAMS II record. 

e Whether the supervisor must meet with the employee to discuss the disposition, as a meeting 
is not required for “Cancel” and “No Action” dispositions. 





’ The correct disposition for these SAIs should have been “No Action — Note for Annual Review Only” or “Cancel.” 
The OIG concluded this based on examination of the Action Item History Reports and the officer TEAMS II reports, 
as well as discussions with the supervisors generating the SAIs regarding their intent. 

$ The supervisors advised the OIG that since the employees evaluated had highly satisfactory performance and 
needed no corrective action or special monitoring, the SAIs should have been assigned a disposition of “No Action.” 
? These SAIs were either created in error or for testing purposes, so the disposition should have been “Cancel.” The 
OIG concluded this based on examination of the Action Item History Reports and the officer TEAMS II reports, as 
well as discussions with the supervisors generating the SAIs regarding their intent. 

10 The SAI was created to monitor a complaint against the employee already initiated 14 days before the SAI was 
generated. Thus, the complaint was not initiated as a result of the SAI, but was preexisting when the SAI was 
created. The narrative for this SAI as a result of the monitoring indicated that no further action was necessary, so 
“No Action” should have been the disposition assigned to the SAI. 


Training Evaluation and Management System (TEAMS) II Audit, Phase II 
Page 9 of 9 
1.0 


For example, if the SAI is incorrectly assigned a disposition of “Special Evaluation Report” 
instead of “No Action — Note for Annual Review Only” (as was the case for three SAIs), the 
disposition will appear on the employee’s TEAMS II record for three years instead of one year, 
and it could give a reviewer of the employee’s TEAMS II record the impression that the 
employee needed special mentoring and monitoring, when in fact he/she did not. In contrast, if 
the SAT is incorrectly assigned a disposition of “No Action” instead of “No Action — Note for 
Annual Review Only,” the only consequence is that the disposition will not appear on the 
employee’s TEAMS II record at all, when it should appear for one year. 


Actions Taken by TEAMS II Development Bureau 

TEAMS II Development Bureau indicated that, per the recent system modifications, the only 
available dispositions for TAIs and PERs are “Complete” and “No Action — Note for Annual 
Review Only,” respectively (see Objective No. 2). 





CONCLUSION 


The OIG found that SAIs were generally not being generated when required. Furthermore, when 
they were being generated, the assigned final disposition in preparing to close out the SAI 

(e.g., “Comment Card,” “Training,” “Commendation,” “No Action,” etc.) was generally not 
adequately justified by the supervisors and managers. Fortunately, this was already recognized 
by the Commanding Officer of TEAMS II Development Bureau and the Bureau has put steps in 
place to address these issues. The OIG did find that for all of the applicable SAIs, there was 
evidence that the prescribed follow-up action was taken. Furthermore, for 36 SAIs, the OIG 
reviewed each employee’s TEAMS II record for any indication of “at-risk” behavior, and based 
on that review, no significant concerns were identified. 


As previously mentioned, the Commanding Officer of TEAMS II Development Bureau advised 
that she concurred with, and had already been aware of, most of the issues raised in this Audit, 
and accordingly had been working over the past several months to modify the system. One 
major system modification already implemented Department-wide is that action items for 
transfers/promotions and annual performance evaluations are now automatically generated by 
TEAMS II, instead of generated manually by supervisors. The OIG believes that this 
modification will greatly improve the Department’s ability to comply with Consent Decree 
Paragraph 47. 


We wish to thank the Commanding Officer and personnel of TEAMS II Development Bureau for 
their cooperation and prompt feedback provided to the OIG during this Audit. 


